REMOTE, AUTOMATIC REPORTING SYSTEM AND METHOD 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

[0001] This invention relates to a system and a method for reporting and tracking one 
5 or more parameters such as time or sales, or for placing orders, etc., especially where 
the person or organization that creates the report or order is remote from the person or 
organization that is to receive and process the report or order. 

DESCRIPTION OF THE RELATED ART 
IN^ 10 [0002] Much of ones work life is tracked and measured. Time cards, time sheets, 
S time clocks and time-tracking software, for example, are features of the modern 
5 workplace that are seemingly as unavoidable as the hierarchies that require them, 
y Consequently, following a routine repeated daily by millions in almost every 
]^ industrialized country, those whose time is to be reported complete a report such as 
;f 15 some type of card or sheet or physical or on-screen form. The report is then passed to 
Q one or more other people, such as a payroll or billing clerk, a project manager, etc. The 
2 data on the report is then usually entered manually into some type of bookkeeping or 
p similar tracking software. 

[0003] This conventional process is, however, as rife with possibilities for error as it is 
20 widespread. For example, every time someone must enter information, be it numerical 
or textual or both, manually onto paper, there is the possibility of error, since someone 
else will usually later have to read and interpret what has been written. The likelihood of 
error is often lessened when using time cards that are punched by machine, but even 
then the chance of error is not eliminated, because as soon as the data on the card is 
25 entered manually into a tracking system, there is a risk of misinterpretation or simply 
just typing errors. Moreover, even if everyone in the chain of reporting were infallible, 
the process itself contains an unavoidable inefficiency whenever reported data must be 
reentered into a tracking system - the very need for data reentry requires both time and 
an employee to do the reentry. 
30 [0004] One way to reduce the need for data reentry would of course be to have the 
employee enter data himself into the tracking system. The disadvantages of this are. 



1 



however, plain. First, the employee may not be comfortable with or competent in using 
the time-tracking software, especially if the software is based on a web browser 
Second, even assuming that every reporting employee has convenient access to a 
suitable computer terminal, it would be wasteful to load multiple copies of the time- 
5 tracking software into each terminal. The alternative of having several employees 
having to share a single instance of the software to report their time, for example, late 
[0005] Because the modern world is full of reports and orders, the problems identified 
above in the context of reporting time arise in other contexts as well, where other 
parameters than time must be reported instead of, or in addition to, time. For example, 
iM 10 employees must often file expense and travel reports, or sales reports, 
jp [0006] Moreover, parameters such as time and expenses are not the only types of 

information that frequently need to be reported. Employees or affiliates may need to file 
reports concerning the progress of projects, or the status of inventory, for example, or 
vendors may wish to send bills quickly in order to speed up the payment process. 
15 [0007] Furthermore, similar problems occur even in contexts where the one reporting 
information is not in any way associated with the recipient of the report. This will often 
occur in the case of processing of customer orders. As is well known, one option 
O offered by today's technology is on-line ordering via the Internet. However, not ail 
companies have, or wish to have, a web site, and not ail potential customers are 
20 comfortable with the thought of placing orders over such an insecure channel. 

Moreover, high-tech, on-line ordering and order processing are not always appropriate. 
For example, it would in most cases be too time-consuming and tedious to expect 
several co-workers in an office to have to go on-line and fill in browser-based forms 
simply to place advanced orders for lunch at one of their favorite restaurants. 
25 [0008] What is needed Is therefore a system for reporting information such as time 
worked, expenses, inventory, orders, etc., that is easy to use, that does not require the 
user to have specialized technology available, that allows for easy verification, that 
eliminates the need for manual data reentry, and that is readily adaptable to many 
different reporting contexts. This invention provides such a system, and a related 
30 method of operation. 
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SUMMARY OF THE INVENTION 
[0009] The invention provides a method, and a corresponding system implementation, 
for collecting reports, such as time sheets, expense reports, order forms, etc., of at 
5 least one parameter, such as time, expenses, order information, and so on. A central 
server is included for automatically receiving from any of a plurality of senders, via a 
transmission channel, an electronic representation of an image of a physical form. The 
form has a plurality of data fields, each corresponding to an indicator of at least a partial 
value of at least one of the parameters. Software modules are included within a form- 
ic 10 processing application for automatically identifying the location of the data fields in the 
g received representation of the image of the fomn; for automatically extracting from the 
5 identified data fields the at least partial values of the corresponding parameters; and for 
m automatically storing the extracted values in a predetermined format in a memory for 

lid 

S subsequent processing. 

;f 15 [0010] A sender, for example a reporting employee or ordering customer, preferably 
O causes the electronic representation of the image of the physical form to be generated 
J2 by using a conventional facsimile machine to transmit the report form. This allows the 
O transmission channel to be a standard telephone line. 

[001 1] The stored, extracted values of the reported parameters may also be 
20 transferred to an external recipient via a network. All processing of the physical form 

after transmission by the sender up to and including transfer to the external recipient via 

the network thereby takes place automatically; this avoids all need for manual data 

reentry and other intermediate manual processing. 

[0012] Each data field in the report preferably indicates a quantifiable or itemizable 
25 value of a corresponding one of the parameters. Nonetheless, other non-quantifiable 
and non-itemizable entries such as a signature may also be included on reports. In this 
case, the central server preferably also stores the received electronic representation of 
the image of the physical form in the memory so that even the non-quantifiable and non- 
itemizable entries by the user onto the physical form are made available for subsequent 
30 review. 
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[0013] The form-processing application preferably also includes a software module 
that receives annotations from intended recipients of the form and stores these 
annotations in the memory along with the stored extracted values of the respective 
received form. 

[0014] The form processing application preferably identifies the location of the data 
fields automatically by performing the following sub-steps: storing an electronic 
representation of a template of each of a plurality of physical forms; automatically 
identifying each received form by performing a best-fit comparison of each received 
electronic representation of the image of the corresponding physical form with the 
stored electronic representations of the templates; automatically registering the received 
electronic representation of the received physical form image with the best-fit electronic 
template representation; and matching the data fields in the received electronic 
representation of the received physical form image with corresponding data fields in the 
best-fit electronic template representation. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0015] Figure 1 is a block diagram that illustrates the main hardware and software 
components of a basic embodiment of the system according to the invention. 
[001 6] Figure 2 illustrates one of the many types of forms that may be used to report 
information using the invention. 

DETAILED DESCRIPTION 
[0017] Figure 1 is a block diagram of the main hardware and software components of 
a basic embodiment of the invention. In this embodiment, it is assumed that a sender 
100 is to transmit some kind of report to a central system 400 ("central server" or simply 
"center"). The sender does this by recording the information to be reported on a form 
200, which is then sent using a transmission device 120, via a channel 300 to the 
central system 400. Upon receiving the transmitted form, the center then processes it in 
order to extract the data from the form and, after optional further processing, store it in a 
data base. Before the individual hardware and software components of the invention 
are discussed in greater detail, these terms and concepts are first explained. 
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[0018] The sender 100 will be any person or group that needs to report information to 
the center 400. Note that this invention may be used with any number of senders. In 
the most commonly anticipated use of the invention, the sender will be an employee 
who needs to report time worked, or some other parameter such as travel expenses, a 
vacation request, sales, inventory, etc. or any combination of such parameters. The 
sender may, however, also be someone not directly associated with the center, such as 
a vendor submitting a bill or a customer submitting an order, such that the center acts 
as an intermediary. 

[0019] The transmission device 120 is preferably a conventional facsimile ("fax") 
;M^10 machine. There are several advantages to using a fax machine. First, whereas not all 
workplaces have convenient Internet connections, almost all have at least one fax 
machine. Second, almost every sender in the modern work environment will be familiar 
!W with the use and operation of a fax machine. Third, the channel 300 used by a fax 
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machine is usually the public telephone network, which allows not only for a dedicated, 
15 point-to-point connection with the center 400, but also is generally more reliable than a 
non-dedicated network such as the Internet. Fourth, all modern fax machines provide 
an easily understood, immediate confirmation of successful transmission or of failure, 
p Fifth, sending fax machines also usually transmit identifying information such as the 
telephone number from which they are transmitting. This makes it possible for 
20 recipients, such as the center 400, to identify the source of a transmission. 

[0020] It would also be possible, however, to use the invention with other transmission 
devices than a fax machine and other connections than a telephone line. One possible 
alternative, for example, would be to use a conventional computer to which a scanner is 
connected. The sender could then scan in the completed form he wishes to send, then 
25 transmit the file in some predefined fomnat to the center, for example as an attachment 
to electronic mail ("e-mail"), directly as a bit-mapped file using fax-modem software, etc. 
In this case, the transmission channel 300 could involve a network other than, or in 
addition to, the telephone system, such as the Internet or a dedicated network (an 
"intranet") within an enterprise. Note that a fax machine itself embodies a scanner, and 
30 as such acts primarily as a combination of a scanner and a modem. All that is required 
to use the invention is some device that allows a physical form to be converted into an 
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electronic representation that can be transmitted to the center 400; moreover, the 
transmission channel 300 may be wired, such as a telephone line, or wireless. 
[0021] The form 200 may, likewise, be of many different types. In the most common 
use of the invention, the form will be a conventional paper form (one or more sheets) on 
5 which the parameter(s) to be reported is marked in some visible manner, along with 
optional associated information such as the sender's identification, signature, etc. Each 
parameter to be reported is preferably amenable to quantification or itemization on a list 
in order to improve the system's ability to interpret its value correctly and 
unambiguously. For example, time can be reported in hours and minutes; inventory 
imo can be reported by marking an item number and a quantity; orders can be reported by 
p marking a box next to the name of the desired items, along with a quantity; performance 
;Ff can be itemized as "excellent," "good," "acceptable," "unacceptable," etc. 
m [0022] Handwritten or printed comments, on the other hand, or signatures, cannot be 
itemized or quantified as readily, if at all. As is discussed further below, however, the 
15 invention allows such information to be Included as well, although it will typically not be 
h processed in the same way as infdmriation that can be quantified or itemized. 
2 [0023] The sender 1 00 will usually be remote from the center 400. Note that the term 
P "remote" does not necessarily imply distance, but rather, in the context of this invention, 
simply that the form is converted into an electronic representation and is transmitted 
20 over some channel to the center in some way. 

[0024] In the simple embodiment of the invention shown in Figure 1 , the center 400 is 
the recipient of the transmitted form 200. This would be the case, for example, where 
the center is in the payroll department of a large company and is used to process time 
sheets sent in by off-site employees. It would also be possible, however, for the center 
25 to act as an intermediary for other parties who are the ones primarily interested in the 
data extracted from the forms transmitted from different users. Extracted data can then 
be transmitted (according to a known schedule or in response to a request) to the other 
recipient parties in any conventional manner, for example in the form of a summary, a 
detailed report, one or more electronic files in a chosen formal, etc., or any combination 
30 of such methods. For example, Figure 1 shows one recipient 500, which communicates 
with the center 400 in any conventional manner via a network 700 such as the Internet. 
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Parties to whom the center-collected data is transferred may then process this data in 
any way they see fit, from a simple review of a report to a full-scale analysis of the 
reported parameter (such as time worked) using other applications. 
[0025] Figure 1 also shows the preferred structure of the center 400, which may be 
5 configured as a conventional computer system controlled by a system administrator 
401. As in other computer systems, the center 400 includes system hardware 402 with 
at least one processor (PROC), system software 406, such as an operating system 
(OS), device drivers, etc., and one or more memory devices 440, which may be any 
combination of high-speed, volatile devices such as RAM memory and mass storage 
!No devices such as disks. 

O [0026] The center also preferably includes a conventional network connection and I/O 
device 410 such as a modem and/or network interface connector (NIC), depending on 

y the type(s) of transmission channel(s) 300 used to send forms. As is well known, the 

M 

.7^ connection device 410 serves as an interface between the channel 300 and the system 
15 hardware and software and transforms a data stream suitable for transmission over the 

U 

O channel into a corresponding data stream that can be processed within the center, and 
if! vice versa. The function of a modem and NIC is well known and is therefore not 
O described further here. 

[0027] In order to convert the transmitted form into data suitable for processing, the 
20 center 400 includes an image capture module 420. This module may be implemented 
with existing technology, and converts the data stream received by the modem/NIC 
component 410 into a file containing a digital representation of the image of the sent 
form. Known fax-server software performs this function, for example, by receiving a fax 
and converting it into an image file in any of several standard formats, such as TIF, GIF, 
25 JPEG, HTML, or a standard bit-map. All that is needed according to the invention is for 
an image of the transmitted form to be generated such that the various data fields of the 
form can be located and interpreted. As Figure 1 illustrates, the output from the image 
capture module 420 is preferably stored directly in the memory 440 for later retrieval of 
the raw image as received from the sender. 
30 [0028] As everyone who has ever received a fax knows, the faxing process often 
skews and warps the document faxed, and noise and inaccuracies often corrupt the 
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image further. In addition, most modern fax machines deliberately compress the 
transmitted image in order to make room to add a header. In essence, a form 200 can 
be considered to consist of an outer boundary, within which information is made visible 
in different fields, that is, portions of the form. The faxing process can distort not only 
the outer boundary, but also the absolute and relative positions of the fields, as well as 
their shape. Before data can be extracted from the various fields of the form, it is 
therefore necessary to identify where the fields are. 

[0029] The form processing module therefore includes a sub-module 431 , which 
detennines the geometry of the received form image. Any known registration algorithm 
may be used to perform this function, and there are several commercially available 
image analysis routines that are suitable for use in the invention. Many algorithms are 
known, for example, that locate certain predetermined geometric shapes such as 
rectangles. 

[0030] Other algorithms are more general, and match a given image with one or more 
templates. This is the preferred algorithm in the invention, since it does not restrict the 
layout of the forms as much as simple shape-location routines. Typically, these 
algorithms compare the test image (in this case, the received image) with a template 
and then uses known mathematical techniques to find a match that is the "best" in some 
predetermined sense, which is usually the minimization of an "energy" or "cost" function. 
For example, if each image (the test image and the template) is represented as a matrix 
of picture elements ("pixels"), then the pixel matrices can be compared element-by- 
element, for example, by forming the absolute value of the difference in their gray-scale 
values. For binary, bit-mapped comparisons, a similar result can be obtained by 
performing a logical "NOT XOR" operation. The test image and template are then 
mathematically (usually through changing the indexing) translated, rotated, and, in the 
most advanced algorithms, warped (similar to a technique known as a "rubber sheet 
transformation"), relative to each other until the sum of the absolute difference values 
(either as is or squared) is a minimum. 

[0031] These algorithms are mathematically similar to holding the form and the 
template up to a light on top of each other, and then, while looking through both, shifting 
them until they appear to "line up." In short, the geometry module 431 perfomis a 
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registration of the received form image with one or more templates, which are pre- 
stored in memory. In order to aid registration, the forms may be provided with one or 
more registration markings. The process of registration can then often be speeded up 
by using the markings as guides. If several markings are provided at different locations 
over the surface of the forms, with known separations and orientations, then the 
markings may also be used to compute local "dewarping" factors that can be applied to 
the pixels in a region near the markings. 

[0032] In the simplest case, there will be only one template that all transmitted forms 
correspond to. In some applications of the invention, however, there may be many 
different types of forms, for each of which there is a corresponding template. One way 
to detemriine which template is to be applied is to establish a different telephone number 
(or network address) for all senders who are using the same form; the telephone 
number to which the form is sent then also identifies the proper form template. Another, 
automatic method is to compare each received form image with all the templates, only 
one of which (presumably, the correct one) will minimize the differences with the form 
image. Yet another method would be to use the identification feature of modern fax 
machines: The standard transmission protocol of a fax usually includes the telephone or 
station numbers of the sending and receiving devices; if specific templates are 
associated with certain sending devices, then the number of each sending device 120 
can be used to select the appropriate template. The form processing module 430 
therefore includes a form identification sub-module 432 that implements any known 
routine, such as those described, to determine which form has been received from a 
sender. Note that the form identification sub-module may be eliminated if only one 
form is ever used, or it may be incorporated into other sub-modules. 
[0033] Regardless of the chosen registration and form-identification routines, once the 
received form image has been registered with a template, the locations of the different 
data fields will be then also be known. The actual data contained in the different data 
fields can then be extracted, also using known technology, in a data extraction sub- 
module 433. Several methods for data extraction are known as "optical character 
recognition" (OCR), which typically recognizes alphanumeric symbols and converts 
them into corresponding ASCII (or equivalent) character sets; "intelligent character 
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recognition" (ICR), which typically also attempts to recognize non-standard fonts and 
even handwriting; "optical mark recognition" (OMR), which detennines whether a data 
field (such as a check box, rectangle, circle, etc.) is filled in or blank; and various 
combinations and variations of these basic concepts. 
5 [0034] Once the data on the form has been located and extracted, it will normally 
need to be analyzed in one or more ways. The analysis will also often depend on which 
entity or entities are to be the ultimate recipients of the result of the analysis. For 
example, if the form is a time sheet, then the analysis may be different for exempt as 
opposed to non-exempt employees, that is, those for whom overtime must be paid and 
those who are on a fixed salary regardless of hours worked. Hours worked each day 

Q 

p during a pay period will, for example, normally need to be summed, or hours worked for 
5h an entire period may need to be associated with different projects. The form processing 
y| module therefore includes a data analysis sub-module 435 that is programmed using 

normal techniques to perform whatever analysis is required in each case. 
!f , 15 [0035] Depending on the circumstances, it may also be desirable to be able to add 
iD notes or othenAirise edit the form data. A corresponding processing sub-module 436 

may therefore be included to allow this. For example, if an employee is reporting time 
p taken as vacation, and that employee has exceeded his available vacation time, then a 
notation to this effect may be made in the data file in order to flag this for a manager. 
20 One or more parties in addition to the sender (in particular, a recipient 500) may also 
wish to add notations or comments to form data, in particular, to data once it has 
already been stored in the memory 440, as is indicated in Figure 1 by the direct 
connection between the sub-module 436 and the memory. In such a multi-party 
embodiment of the invention, parties 500 may contact the center 400 and, after 
25 authentication using a password, add notes to the file into which a form is converted. 
These notes may be textual, but preferably also include notes in the fomi of attached 
audio files, for example in the widely used ".wav" fomnat. 
[0036] Once a form's data has been extracted and analyzed as needed, it will 
normally need to be stored in a format suitable for later retrieval and further analysis 
30 and annotation. Different parties may also request data storage in different formats. 
One party may, for example, want to be able to retrieve data into a spreadsheet such as 
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Microsoft Excel whereas another may want to retrieve data In QuIckBooks format. In 
order to allow the center to adapt to the needs of external parties, rather than forcing 
them to accept a fonnatting decision used in the center, the form processing module 
430 according to the invention preferably includes a format conversion sub-module 437 
that converts the data from whatever format is chosen to initially represent extracted 
data (preferably, a generic format) to any of one or more predefined formats before 
storing the data in memory 440. 

[0037] Alternatively, all data could be stored in memory in a format chosen to meet 
the needs or preferences of the center 400, in which case the format conversion sub- 
module could be invoked to reformat stored data only when it is to be exported to 
another party. Note that in the case where an external recipient (such as recipient 500 
illustrated in Figure 1) receives form data from the center via the Internet, this means 
that, once the sender transmits a form 200, preferably via fax, all processing is carried 
out automatically - with no need for manual data reentry or other human intervention - 
even to the point of transferring the data via the Internet for subsequent processing. 
[0038] The sub-modules 431-437 may ail be implemented using known programming 
techniques with conventional hardware support as needed in any given application. 

[0039] Figure 2 illustrates one example of a possible form that could be used in the 
invention, in practice, of course, each user of the system will usually have unique 
requirements and specifications for the form(s) they will use. The invention is able to 
accommodate any forms that, as is mentioned above, have data fields in which possible 
values of parameters to be reported can be quantified or itemized. 
[0040] The form example shown in Figure 2, however, resembles in structure a fonri 
used successfully in tests of a prototype of the invention, in that it allowed for accurate 
data extraction when transmitted from several different conventional fax machines. As 
the various headings indicate, this example fomn is set up to enable reporting of time 
worked over the course of a seven-day week. In this example, hours worked each day 
are indicated on a row of entry fields, with columns for the day, as well as for the 
parameters "Regular Time" and "Overtime." For the sake of clarity, the data entry rows 
and columns are shown larger than they were on the test form used in the prototype of 
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the invention. Moreover, the rows for data entry for the days Tuesday through Saturday 
have been omitted from Figure 2, because of space constraints and again for the sake 
of clarity. The actual layout of the fomi will depend on the needs of the users; the 
invention does not require any particular layout or headings. Note, moreover, that a 
5 form may consist of more than one page. 

[0041] The illustrated form includes several fields that are preferably pre-printed when 
the form is delivered to the sender (for example, at the beginning of a pay period for 
which he is to report time worked). Examples of such fields are fields for a logo, general 
information and instructions. 

iM'io [0042] A field is also shown for the sender's signature. This will of course usually not 
be readable by any recognition software, but this is not necessary according to the 

;S invention. Rather, recall that an image of the sent form is preferably stored as is in the 

W memory 440. If a user (either the administrator 401 , or an external party such as a 

hi 

,g payroll clerk or manager at the sender's employer, etc) later wishes to verily that the 
;f 15 form was properly signed, then the user will be able to retrieve not only the analyzed 
ig data file, but also the original image, on which the signature will appear. This is of 

course also the case for any other visible feature of the form. 
jO [0043] When using the illustrated form, the sender enters data by darkening small 
data "boxes," each of which is a corresponding data field. Thus, any amount of time 
20 can be indicated, in five minute increments, by darkening the appropriate boxes, for 
example, using a pencil or suitable pen or marker. The sender similarly indicates such 
information as what his employee ID number is. In Figure 2, the sender has thus 
marked that he is employee C4 and that, on Monday, he worked exactly eight regular 
hours and one hour fifteen minutes of overtime. His weekly total was 40 regular hours 
25 and two hours fifteen minutes of overtime. 

[0044] The illustrated form also shows a block "Time Period." This may be preprinted, 
or it, too, may be designed with data boxes that the sender can darken; this would allow 
copies of a single form to be used over many time periods. 

[0045] The form shown in Figure 2 also illustrates how registration marks, such as 
30 marks 202, 204, may be included on the form as aides to image registration and 

location of the data entry fields, that is, the data boxes. Another field (not shown) may 



12 



also be included to identify the form and perhaps even the entity (for example, 
employer) associated with the report; for example, when the form is created and 
delivered to the user, a bar or other standard code that Is easily decipherable by 
software could be included in a form field to aid form identification. 
5 [0046] As part of the data extraction process performed by the sub-module 433, for 
example, the optical mark recognition, the form processing system according to the 
invention will sense which boxes the user has darkened. Because the form will have 
been registered with a template, the darkened boxes can be readily associated with the 
correct data values for the different parameters to be reported. The data analysis sub- 
|j=^lO module 435 may then also carry out such computations as calculating the total hours 
•q reported as worked, which can then be stored along with the other form data in memory, 
p The calculated total can then be checked against the value the sender himself has 
y entered. If the calculated sum does not match the sum the user entered in the 'Total 
Hours" block, then an annotation to this effect can be made in the stored data file and 
^ 15 the administrator can be notified that the form is invalid because of at least one 
Q identified error. The sender can then be instructed in any known manner to send a 

correctly completed form, 
p [0047] Tests have indicated that "boxes" such as those illustrated in Figure 2, with the 
values shown within each respective box, were easy to use and understand, and could 
20 be recognized by software with very high reliability. Other types of data entry fields may 
of course be used instead. For entering time, for example, two small clock shapes 
labeled "hours" and "minutes" could be printed on the form; the sender could then draw 
hour and minute "hands" in the respective clock figures to indicate time. A single clock 
figure could be used instead, on which the user draws in both hour and minute "hands"; 
25 to aid in the distinction, an inner ring could be included to indicate the proper length of 
the hour hand, the outer ring of the clock indicating the proper length of the minute 
hand. 

[0048] As yet another alternative for entering numerical data such as how many of a 
particular item the sender wishes to order (or report), the form could include data entry 
30 boxes for units, tens, hundreds, etc. Each box could then include light, dotted or 

dashed lines oriented to correspond to the segments of a conventional seven-segment 
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display. By darkening the appropriate segments, the sender can "print" the numbers in 
an unambiguous, handwriting-independent manner. Such data entry boxes have been 
used successfully for many years for automatic optical reading of postal codes on 
mailed envelopes in several Eastern European countries. 
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